Extracting Structured Supervision From Captions for Weakly Supervised Semantic Segmentation

Weakly supervised semantic segmentation (WSSS) methods have received significant attention in recent years, since they can dramatically reduce the annotation costs of fully alternatives. While most previous studies focused on leveraging classification labels, we explore instead use image captions, which be obtained easily from web and contain richer visual information. Existing for this task assigned text snippets to relevant labels by simply matching class names, then employed a model trained localize arbitrary images generate pseudo-ground truth masks. Instead, propose dedicated caption processing module extract structured supervision consisting improved object their attributes, additional background categories, all are useful improving quality. This uses syntactic structures learned data, relations retrieved knowledge database, without requiring annotations specific domain, consequently extended immediately new categories. We present novel localization network, is only these labels. strategy simplifies design, while focusing training signals Finally, describe method types maps obtain high-quality masks, used train model. On challenging MS-COCO dataset, our moves state-of-the-art forward significantly WSSS with image-level margin 7.6% absolute (26.7% relative) mean Intersection-over-Union, achieving 54.5% precision 50.9% recall.